Models Estimation for Incomplete Paired-Comparison Experiments Based on Least-Squares Solution

Incomplete paired comparison is an important scaling technique in vision science since the total number of paired comparisons for n stimuli is n(n-1)/2 which becomes prohibitive for large values of n. However, the experimental designer often struggles with questions such as what is the smallest limit for the proportion of paired comparisons included that will still allow reliable estimations of scale values? Monte-Carlo computational simulations were previously carried out using a model of an ideal observer. The results showed that the proportion of paired comparisons that is included is more critical than the number of observers who make those observations. This work aims to test the results from the computational simulation with 25 real observers and 10 stimuli from the gray scale. The psychophysical data suggest that when each observer evaluates the same pairs, accuracy of the derived scale values increases with the proportion of pairs evaluated and the number of observers; the proportion of pairs is, however, more critical and this agrees with the results of the simulation. The psychophysical data also suggest when the each observer estimates a different pairs (albeit with the same proportion of pairs being evaluated) the accuracy of scale values does not always increase monotonically with the number of pairs being evaluated.